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Current curricalua evaluation models tend to cater to 
desirable or ideal situations and offer little help to the evaluator 
of ongoing curricular innovations, nany of vhich are characterized by 
little assistance or control of inpleMentation, little Monitoring or 
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■odification or development. This paper describes the conception, 
operationalization, validation, and role of an Arbitrary 
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oriented junior high school science curriculun in Saskatchevan 
operating vithin the context described above. The AIS is included in 
the appendix of this report. (Author/BC) 
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ABSTRACT 



Current curriculum evaluation models tend to cater to desirable 
or ideal situations and offer little help to t e evaluator of ongoing 
curricular innovations, many of v^hich are characterised by little 
assistance or control of implementation, little monitoring or supervision 
of operation, and haphazard pos t-iinplementation modification or develop- 
ment . 

This paper describes the conception, operationalisation , validation, 
and role of an Arbitrary Implementation Scale in an ex post facto curriculum 
evaluation model V7hich facilitated the province-wide assessment of an 
inquiry oriented Junior High School Science Curriculum operating v/ithin the 
context described above. 
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BACKGl^lOUND TO Tin.-: STUDY 

Introduction of the Curricu lum'^ 

Following a four year curriculum devftlopnient period involving writing 
conferences, trials, pilot projects, and revisions sinnilar in pattern to 
that which Grobman ' (1970 , p. 4) describes, a new junior high schoo? science 
curriculum was introduced in Saskatchewan schooJ.s in two phases in 1968. 
Firstly^ existing pilot projects were expanded; with in-service education 
being accomplished through teacher interaction and cooperation. The second 
phase involved province-wide half-da^' workships, led by teachers trained by 
the Province's Science Curriculum Committee during a two day workshop. Sub- 
sequent implementation, in-service education, and local adaptations were the 
responsibility of local superintendents and their staffs within approximately 
sixty school units covering the southern third of the province - the northern 
two-thirds of Saskatchewan with its remote settlements, fly-in centres, and 
native Indian population is the responsibility of central government agencies. 
The General Evaluation : 

Need : In 1972 the authors prepared a research proposal for the evaluation of 
junior high school science in Saskatchev;an and commenced discussions with the 
Provincial Science Education Coininittee . The following reasons underlined 
the need for the study: i) By 1970 science education had undergone two 
decades of curriculum reform both in Britain and North America, ii) New 
thinking related to science teaching was creating pressures for change, iii) 
The curriculum had been in operation for four years and represented a signifi- 
cant departure from the previous curriculum. During this time scattered local 
formative evaluation had been taking place v;ith some central initiatives but 



1 

Vho Vi.-udcr ir* rej-crrrid I'o ? ;pA"no7"ri L .ud i:>vcc.i}i(' ohjv.cA'iv^:r> cf Li»e 
currlculuia contained in t.lir^ /^j'jv.-ud:'' x * 
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not of -ufficient magnitude and depth to be called a continuous prograra of 
evaluation. Certainly not sufficient enough to provide an ecological picture 
of the degree of iiiiplcmentation, province-wide, nor sufficient enough to give 
feedback on student and other criterion measures. It was hoped that this 
project would provide evaluative data for decision-making and be a spur for 
a continuous evaluation on a province-wide basis, both central and local. 
Purpose : The purpose of the general project was, therefore, five-fo].d: 

1, To assess the degree of imj-»lenK»ntation of the program, 

2, To gain some measure of its suitability and worth, 

3, To determine if such factors as costs, demographic variables 
associated with teachers and school organisation were related 
to implementation, 

4, To compare student outcomes for classrooms that have implemented 
the program with those that had not, 

5, To gain an ecological-'-p i cture of curriculum and instruction for 
junior high school science in the Province of Saskatchewan, 

Settin g: At the time of the evaluation it V7as expected that most junior high 
school classrooms would have introduced the curriculum, although the extent 
of actual implementation would vary considerably due to the following factors: 

i) The junior high classrooms may have been in different school set- 
tings; either elementary school (for grades 7 and 8), junior high schools, or 
high schools (for grade 9) , 

ii) Rooms and facilities would vary considerably, 
iii) Implementation required expenditure for equipment which ranged 
from $500 to $1,000 per class, 

iv) Little guaranteed assi5>tance with iinplementati on (in-service, 
sii])e i"v .i u i on , c on fiu 1. ten L 1 j c 1 p , i' lie ,) ; I !iv.' i ;i 1 1 i . > l i vr. f or 1 1 ; I .s w on 1 d 1 ' 
local prcrogr; u lvl; of superintcrdcriL , princlp:; I.s , and school staff 

Q 1, See Barker, R,L. P^cological Poychology, Stanford CaMf..: Stanford University 

Press, 1968. 
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j) Minimal centiral control over implementation procedures and 
pattern allowing for local adaptation. 

vi) Minimal monitoring or supervision of curricula in operation, 
vii) Haphazard or broke;i-f rent post-iinplGmentation patterns in 
modification and development. 

It was suspected that these factors would result in a rather mottled 
ecological picture of implementation throvighouc the Province. 

Eyalu.1t i on Strategy and Model : In essence, the general problem for the total 
project was to develop a strategy and model for evaluat?.ng this curriculum, 
serving the purposes stated previously, and having implementation constraints 
and variations describc^d above. Limited financial resources for the study in 
a Province with a population sparsely distributed throughout vast areas also 
had to be borne in mind. 

The researchers v/erc fortunate to be able to work as independent 
external evaluators when it seemed appropriate but also could involve the 
Provincial Committee, v;liich developed the curriculum, very intimately at 
appropriate points, hopefully harvesting the best of internal and external 
modes of evaluation. This also enhanced the chance of subsequent recommend-- 
ations being accepted and implemented. 

An examination of existing curriculum evaluation models via general 

sources (Taylor, 1972^ Worthetn, 1973) revealed that they were not completely 

appropriate in an ex post fa cto situation although parts could be utilised; 

the same could be said for accepted research designs (Cambell and Stanley, 

1963) - atz VJalbcGscr (1968, p. 54) states: 

"There is no universal research design that can be applied 
unifonr.ly to all curriculum evaluations. In facf: much of the 
'text-book* research design has only limited application to 

the dc-^s 1 p,n of an evn 1 via I i mi for :\n orr-.oi n f\ pro joct" 
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Tyler's (.1951) simple and basic sequence of denoting ob s ervable 
objectives, spec i fyin g learning exp eriences likely to contribute to the 
attainment of the objectives, followed by comparing the outcomes to the 
intended objectives , has been mirrored and elaborated by Provus (1968) in 
his systems approach and discrepancy model. Contributions by Stake (1967) 
with his antecedents , transactions , and outc omes and similarly by Stuff le- 
beam (1971) with the context , input , process , and p roduct (C.I.P.P) model 
as well as Scrivcn's (1967) formative and summativc evaluation have all 
served to discriminate different sets of elements within the domain of 
curriculum evaluation. However, it is seldom that a ready built model will 
suit a particular situation especially when complicated by the factors 
mentioned previously (p. 2); furthermore, a perusal of the literature reveals 
that little has been written concerning appropriate means for conducting ex 
post ^acto evaluation*^ where no facilitative steps were taken during curric- 
ulum development and implementation; so, the researchers were required to 
develop a generalized model for this type of situation from which to \;ork. 

Recent models developed for curriculum evaluation mentioned above 
provide a base from which to work, but were also somewhat inappropriate 
since they were designed to follow programs through the planning, develop- 
ment, implementation and final evaluation stages. Hence only partial use 
could be made of these. 

The model developed for this study utilized the notion of discrepancy 
between expectation and reality. It was assumed that as the program had 
bcvjn in cTfecr. fur f : vt- y , tirae hr.d elapsed to allov; t(V?soTK.bly 



This is understandably so ia that the ex post facto mode is a less 

do;.ji;:blc ior'.r. .'f .'V.: l.i:a(. ' ( ; lu^ v v v r lie: L. -.s , ; . i l i>« < I: J oii^; ari^u: wlicu'e 

i. t if. n o s L u s c ■ f u 1 \-:\ ion c ot n j v » r e to no o vi) 1 u ;.j 1 1 ov . 1 1 1 : 1 o )* r. w o. s 1 1 ou 1 d 

V70 r ! : t o\ r cl s 1 1 \ h c s L ex [; o^: t I'ac J:o i; ! o J e 1 s . 
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stable and observable clnssrooni i»rac Licus to emerge* Furthermore > if such 
practices were occuring according to reasonable expectations, then it could 
be hopcid that 'Mmplementatiou*' had, in fact, been achieved. The degree to 
which a discrepancy existed between expectations and reality represented 
the degree to which implementation had not been achieved. Such a notion 
required the development of a set of expectations and a means of observing 
reality to see if those expectations had been met. Figure 1, overleaf,, 
includes the total model and flow^chart of operationt>, • 

The expectations for implementation were essentially a description 
of thone conditions that ought to exist in classrooms where the program was 
in operation. These conditions should logically follow from a careful 
consideration of the program objectives. Hence, the first stage of the 
evaluation model involved (a) clarification of program objectives'^, and (b) 
definition of expected conditions. 

The second stage of the model involved the development of observa- 
tional instruments. These fell into two broad categories - student outcomes 
and classroom transactions and conditions. 

These outcoiries, transactions, and conditions provided a basis for the 
collection and development of the data gathering instruments which were: 

1. A general questionnaire containing some eighty items designed to 
gather data on facilities, costs, equipment, in-service training, teaching 
activities, teacher background, feelings, opinions, and ^attitudes . 

2. Student measures including an interest inventory, attitude scale, 
cognitive test, and the student *s perception of classroom. 

3. The Science Classroom Observation Form which enabled observers to 
rate interactions in the classroom. 

^4 . A guide- for i n t c r \' i. c • ; L r, r tc:;:c.]\< : • 

r 

See Appendix. 
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I 

"Lhe general straCe^^y of the. study was to utilise, tlie general qui'.s- 
tiuanalre to glean d<ata from all the teachers of jiuiior liigh school scieni.e 
in the Province which would serve purposes 1, 2, 3, and 5 (p. 2) of Die 
study. The instruments mentioned in 2, 3 and 4 above for students and 
observers were designed for use in an in~depth ^study which would serve the 
same purposes as the general questionnaire, Jalidating and/or qualifying 
its findings, but most importantly v/ould serve to provide data for a com- 
parison of stMdent outcomes of classrooms which had implemented the curriculum 
to a large degree with tliose which had not (Purpose 4) . 

We will not take time and space here to examine the detailed develop- 
ment, validation and use of the above instruments as it is documented else- 
where (Butt, 1973b; VJideen, 1973, 1974) and because the specific focus of 
this paper is the Arbitrary Implementation Scale which, finally, vie can discuss. 
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. THJi: ARKlT]l/,Ri% IMrLKMENTA'IJiqN SCALE 

Purpose ; Al^tcr having developed Lhc general evaluation mode] , the strategy inen- 
tioncd above, and n^ade bo^',ianivigs on the development of tlie Instruments, there stilj 
* remained the problem of as.sensing, in some quantitative way, the degree of 
implementation in each classroom, together wi^h a research design problem 
for analysing the data from both the gef\,eral questionnaire and the in-depth 
'Study. 

An Arbitrary Tinnlemontatj on S cale (MS) was proposed to overcome the 
above problems; it was hoped th^^t we could identify a scale of items based 
teacher s rc^ ponser> from the general questionnaire (it was unthinkable to 
visit every classroom in the Province) which would validly indicate degree 
of implementation in the classrooms within which individual teachers 
functioned J and would therefore act as a useful "independent" variable for 
research design and data analysis purposes. 
Development 

Items from the general questionnaire, which had thus already been 
through the preliminary validating (face and content) procedures of the 
judgement of a panel of experts (the research team and the provincial 
committee) pilot trials, and revision, were identified as being key indica- 
tors of successful implementa t ion of the program . The final version of 
the AIS consisted of 28 items which fell into five categories reflecting 
pervasive constructions associated with implementation. These include: 

1* The opportunity and extent of in-service education (two items). 

2. The l: :o.;!cd:^c oT, acceptance and agreement v.dth the philosophy, . | 
aims, and objectives of the curriculum (six items) 

3. The self— perception of teaching ability for the curriculum (5 items) 
^. The cxtcHc Itj ;.!;jc]-i cocl^iin r<u.Lorb h^jlped or hindered in inplc- 

M n L a L i on ' ( t. c n i l c ais ) 
5. Specific practices in teaching and evaluation (four items) 

ERIC 
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TheoC Items were picked because of their ability to discriminate 
degrees of i.mplcmenLation during the validation of the general questionnaire. 
The AIS scale is included at the conclusion of this paper. 

For scoring purpose the items were treated as part of a Likert scale 
and a single score assigned to each respondent. The coding of items 1 and 
2 was such that they were reverse scored. The wording of items 4, 7, 16, 17, 
18, 19, 20, 21, 22, 23, 24, 25, 27, and 28 was such that they were also 
reverse scored. Items were differentially weighted in order that each would 
contribute equally to the composite score. It seemed appropriate that mis- 
sing data be treated as zero in the ,case of items 3, 4, 5, 6, and 7 while 
other items omitted were assigned the overall mean. 

Vali dation : The "a priori'* criteria stated to further assess the validity 
of the AIS included the following: 

1. The MS should correlate with the teachers' perception of how 
well the program had been implemented. 

2. The AIS scores should show significant differences between samples 
of classrooms using the new program and the former program. 

3. The AIS scores should correlate highly with the ratings of the 
research team done on a random sample of classrooms. 

4. The AIS scores should show significant differences between class- 
rooms rated by a science supervisor as having good implementation 
or poor implementation. 

Data from the general questionnaire provided information relative to 
the first and second criteria. The responses to items which asked teachers 
in the three grades to state their opinion on the degree to which the curriculum 
had ])no.n impl omen ted ^ were correlated with scores on the AIS. The correlations 
\v\ rc .. ign ! ficaiit. Cp 1.01, n^-ulS) v/itli values of jj^^, Ji7_ and 'j4^ resp<u'. ti j y 
for ^;^venLh. eight and ninth grade teachers. 
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The pccond cT*i tcri-ci statr^d that tlie AIS should show differences 
between diff extent groups In the sample. Means for different groups clas- 
sified according to grade and pro<;;ranis taught arc provided in Table I. 
It can be seen that differences do exist. For example, "Life "cience 7" 
group are approximately half a standard deviation above the "Earth 8" group. 
This was expected since evidence pointed to a better implementation picture 
in grade seven than the eighth and ninth grade. Teachers of the "IPS" 
group, a highly laboratory oriented program, exceeded all others by nearly 
one standard deviation. Of most significance is the mean of the " Science 
Activities " (former program) which is well below all other groups. 

The data provided in Table II is related to the third and fourth 
criteria. As indicated earlier the three member research team independently 
ranked a random sample of completed questionnaires according to degree of 
implementation « This v/as done on the basis of a subjective study of each 
questionnaire witliout prior consultation as to prior criteria for ranking, 
and v*7ithout prior knowledge of AIS items. The AIS scores from these ques- 
tionnaii^es v;ere later determined for the upper and lower ten. The mean 
difference between these groups as shovm in Table II is highly significant. 

The data for the second comparison shouTi in Table II was obtained by 
selecting a school system where variation in program implementation existed. 
The superintendent was then asked to indicate classrooms which had implemented 
the program an.l those vhich had not. It can be seen that the difference 
between the two groups is significant. 

A reliability of 0.78 (K.R.) was obtained for the AIS scale; On the basis 
of the steps taken during the development of the general questionnaire and 
subsequently the AIS together with the results reported in this section, the 
invc-^s L i j/n ("c:; rs f.clt lli.il, i h^ AIS v/aL- a reasonably reliable and valid nensure 
f:or u;ie as an indicator of program iuiplementatioa. 
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TABLE I 



VALUES OP MS SCORES FOR DIFFERENT GROUPS IN THE STUDY 



GROUP N MEAN SD 

Grades 7, 8 & 9 28 110.54 16.56 

Life science 1 152 112.28 16.19 

Earth 8 54 10j.69 14.60 

Space 9 74 104.36 12.02 

Science Aclivitics 7 & 8 

(former program) 17 97.33 12.63 

IPS 27 121.15 12.89 
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Collcc • ^ ^^iL_^'L.J?iL^i} • ^^^^ fi.ri-;t survey phase of the study, the general 

quostionnai re wa-; distributed to every Leachcr involved in the teaching of 
•jun ior-hi^;h school sci<-nice'^ in the Province via the local superintendents. 
The returnr^ v;ere mailed directly to the researchers by the respondents. Of 
approximately 960 junior-high school science teachers who received the 
questionnaire 612 returned a completed version representing a return rate of 
6A%; a very high rate for this type of survey and length of questionnaire. 

Subsequent to the return of the general questionnaires the second 
in-depth phase of the study was conducted with* a stratified random sample 
constructed on the basis of degree of implementation as judged from the 
general questionnaire. The sample had to be limited to 43 classrooms involving 
1165 students due to limited resources. In each classroom all students 
responded to Attitude, Interest, Cognitive, and Classroom Perception measures; 
the science classroom interactions were observed and rated using the Science 
Classroom Obsei"vation F(mn, and the teacher was interviewed regarding aspects 
of the cur?'Iculuin, its development and implementation. 

The comprehensive details of the total evaluation project are reported 
elsev/here (Butt, 1973b; Wideen, 1974). Here v/e will restrict ourselves to 
the data involving the AIS. 



1 

A cOiiijJLi'l uy ::».ihj<.:(t in S«:f;ka Lciu'V/aii . 
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USES OF THE SCALE 

Pes c r .1 p t i vc ^ S t a t: 1 8 1 i c. s ; For the respondents to the general questionnaire 
(n-^'GlS) the MS had a jnunn score of 108.7 with a standard deviation of 15.6; 
the scores ranged from 66 to 155. A frequency distribution of scores can be 
seen below in Figure II. 

Fig^ure II 




AIS 

Tables (VI, VII, VIII) of item means and standard deviations on AIS for all 
respondents^ top scoring half and bottom scoring half splits, as well as 
item correlations with total AIS scores, are included in the Appendix. 
An AIS as a Rating Scale for Cu rr iculum Projects : Cyclic arguments, vjhose 
use is defended by Rozenboom (1966), would allov7 us to include here the 
results and data from the validating procedures of the AIS (p. 9 .) partic- 
ularly with respect to differing degrees of implementation achieved by 
different courses or curriculum packages v/ithin the total junior-high school 
program as indicated by data in Table I. New elements of the program exceed 
I. hf' lomi'.r c'-irri f.'iilui 1 In iinplerncn I n t:ion scoror; 'J.P.S. j:: coring hli'}\esl. This 
suggests that a genera] i^ed implementation scale such as the AIS coald serve 
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as one rating mechanism Cor many of ihc now cuiriculum i)roje.ct:.s (both inter- 
project or intra-project) . 

MS and Correlat iun \j Lth Otlu M: Var in bles ; One objective of the study was to 
investigate the relationships between implementation and such factors as 
facilities, costs, and biographical characteristics of teachers. Table III 
details the correlations between the AIS and these variables. The relation- 
ships or lack of relationships and their implications are too numerous to 
discuss here but the data illustrate the usefulness of an AIS for teachers, 
currJci'lum developers and evaluators, as well as educational decision makers; 
for example, '*In-Scrvice" correlated highly with AIS (.44 p<-01) and while 
only 50% of teachers in the province had the opportunity to attend these 
series. In a sparsely ):)opulated province. Area (Urban - Rural) correlated 
negatively wich AIS (-,26, p<.01) indicating the disparities between Urban 
and Rural areas which must be overcome. Costs, equipment, and space facilities 
relationships with AIS offer support for budget expenditures, in terms of 
implement at i on . 

AIS and S tudent Vari ables; Table IV details the correlations betv;een AIS and 
student and other variables. Again^ these are just Illustrative data and are 
not discussed at length; it is interesting to note significant correlations 
becv;een AIS scores and both observers ratings and student perceptions. The 
other main u.se of AIS in the classroom phase of the study was as a main effect 
in an analysis of variance. The analysis of variance using AIS and SCOF as 
main effects and the student measures as dependent variables is showi in Table 

conducted to determine if there v;ere interactive effects between the AIS and 
SCOF as main effects. 

It can be scon that in only one caii^e .i s an F value signi f i Cou t . Tills 
was where the SPSC scores v;erc dc-pendent variables and AIS the main effect. 
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TABLE III 



CORRELATIO>?S BETV7EEN ARBITRARY IJfl^LE- 
MENTATION SCALE AND SELECTED VARIABLES 



Variable 



Correlation with AIS 



Teacher 

Character! sties 



Tea ching, 
Actlv.i ties 



Sex 
Age 

Area (Urban — Rural) 
University Education 
Academic Science 
Teaching Experience 

Type of Teacher (Senior-Junior High- 
Elementary) 
In Service Sessions (If you had them 

(did they help?) 
Average Class Enrolment 
Number of Science Classes Taught 



Providing Notes 
Field Trips 
Discuss ions 

Student Library Research 
Periodic Quizzes and Tests 
Check Lists and Inventories 
Written VJork (papers, etc.) 
Interviews 

Subjective Assessment of Attiludes 
and Interests 



.06 
.02 

.22* 

.31** 

.07 

.10 

.44** 

.18 

.24* 



-.29** 

.23* 

.10 

.16 
-.12 

.18 
-.04 

.22* 

.15 



Teach ers Fe elings , 
Perceptions , etc . 



Are you aware of Philosophy? 

of Program? (Yes-No) 
Teacher Perception of Degree of 

Implementation of : 



Grade 7 ) 

Grade 8 ) 

Grade 9 ) AIvS 



Validatio n 
Criteria of 



-.22* 



.53** 
.47** 
.35** 



Continued . . . . 
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Coat inued 



Helps in In p 1 o ^ 
menlci f.i.on 



Parentis an 
Students I'ear.tions 



Aviri lab i lit: Y o1: l-^quipnient .48=^^'^' 

Adequate Reference and Library Resources .36^^*^* 

Adn:inistrativG Support ,^1^^ 

Favourable Student Reaction .40'''^^ 

School Board Support .37'^'^ 



Since Implementation Student Interest 

has (Decreased Increased) .37** 

Impact of Curriculum on Parents 

(Unfavourable Favourable) .33** 



FVu^Qi^ic^s^ 

Equ ipinen t , C o trts 



In \n^at T -pe of Room Do You Mainly Teach? 
(Unmodified Classroom Fully Equipped 



Costs of Equipment Available 
Crade 7 
Grade 8 
Grade 9 



Lab) 



.37** 

0.31** 
0.49^* 
0.40** 



Tear her Sumirt a r j s ing 
Opinions of P ar- 
ticular Courses 



Grade 7 
Grade 8 
Grade 9 



(dislike like) 



0.31** 

0.20* 

0.27** 



* p / .03 
** p ^ .01 
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TAHLL-: IV 

CORRELATIONS HETOr>l':>3 MS AKD STUDENT MEASURES, 
AND THE SCri:NCK CLASSROOM OBSERVATION FORM 



Student Interest Scale .01 

Student Perceived Science Classroom . "^j^ rMi^ 

Moore-Sutman Science Attitude Scale .22 

Cognotive Measure -.03 

Grade -.10 
Science Classroom Ob^Jervat: Lon Form 

Environment ) 

) Tlicse are tentative 

Pupil ) .23 

) Subscales 

Teacher ) 35^^ 

Total .24* 



p .01 ) 

^'"^ ]> ,05 ) Utilising the classroom as tlie experimental unit 

) 

*p ^ .10 ) 

^p/..oi ) 

j Utilising the students as the experimental unit, 
-p A .05 ) 
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An (*xa;.* iiia^ i on of the cell means indicated an olovution of scores on tlie 
SPSC for the top scoring lu'ilf MS p/ruup; students felt better able to utilise 
m^iterinls, inquiry d i f;rovci*v^d in hiyli scoring MS cJnssrooms. No other F 
values were significant indicating no effects for other measures and no 
interactive effects. Further analyses v/ere run utilising AIS and grade; no 
differences were revealed in these analyses that liad not been revealed pre- 
viously. It is interesting that jn tlris analysis the Students Forccived 
Science Classroom emerged as being the only criterion ineasure significantly 
related to AIS in view of Anderson's (1969, 1970) v;ork. on climate (see also 
Butt, 1973a) learning, and curriculum. To find even' the one relationship 
vas surprising to the researchers, beari.ng in mind the possible confounding 
factors in the mottled program patterns, especially v^hen only a mid- line 
split in AIS scores could he utLlih:ed for analysis purposes. Student interest, 
attitude, and cognitive abilities were not detected as being significantly 
different for higli and low AIS scores, 

Altf^rnative interpretations for these results include :~ 
i) As just stated, the "treatments" perceived as different by the 
students have no significant effect on other studeiit outcomes 
(Cogni tive. Interest , Attitude) . 
ii) The "treatments" are not sufficiently different to liave a detectable 
effect on student interest, attitude, and cognotive outcomes, 
iii) The AIS scale does not sufficiently discriminate implementation 
for the purposes of this analysis, 
iv) Con fou)) ding variables present obscure "treatment" effects. 
The results suggest that program implementation had been successful in terms 
of affecting students perce})tlons and observers ratings, but as yet has had 
i » t J e eCi'.'ft (Ml r'rli:'r at l' r f h'.it:o:; l:v.:c\:^u:•.(l j:; ::A:udy. 

o 
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"''urthcr x-efjning of the scale-;, selectioa of classrooms from the tvjo 
extremes of AIS ficoriag range, a larger sample of classrooms, with separate 
replica tion:s for each of ^^rMde.s seven, eij^ht, and nine, should guard against: 
ii, iii, and iv above for future use of the scale. 

Other Uses of AIS ; Throughout this report, data generated utilising the AIS 
as an independent variable has served to build up an ecological picture of 
the current state of junior-high school science education in the Province of 
Saskiitchewan . One may even build a visual map of the curriculum ecology of 
the Province utilising AIS or other criteria, drawing contours (if they 
emerge) on the basis of iso-implements (cf. isobars or iso-therms) . 

It has been possible to construct profiles of variables wliere the 
curriculum lias and iias not been iniplemented ; it is anticipated that these, 
and the AIS in general, may be utilised as a backdrop and stimulant for 
^ C'Cal cval nation , cur r i c u lum j nip r o v emeu t , and d eve I o pmen t . 
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SUMMARY, DISCl'SSIOV AMD COKCLUSXONR 

Currcint models In I'he n'.old cjT curriculum evaluation have tended 
towards the ideal siiualiun in terp.'-: of one or more of tlie following: the 
objectives of the evaluation; perscrinel available to conduct the evaluation; 
the scope of the evaluation; and the resources available. 

Iirtiile these conceptually and practically desirable constructs inay 
act as useful inputs prior to curriculuTu development and innovation, for 
providing a sound and co^iiprebensivc evaluation model in school systems v;hich 
have the requisite resources and personnel, there exist many ongoin g innova- 
tive efforts vhich requlT'e directional decision-making on Llie bais of some 
sort of evaluation, for which these models offer very little pragmatic help. 

Key factors v/hich characterise these curricular innovations, wli ether 
by default or deliberate design, are: 

i) Little assistance with implementation 
ii) Little control of jmplementation 
iii) Little monitoring or supervision of curricula in operation 
jv) Haphazard pos t-implep.entat ion modification and development 

These factors can and have resulted in rather mottled ecological 
pictures\^ throughout a large area, province, or state, with regard to varieties 
and degrees of implementation. Tliis presents tJie evaluator with a difficult 
task v/hen existing models of evaljiation and desirable research strategies are 
considered. This is usually compounded (especially latterly) by lack of 
funds and the like, which bar controlled longitudinal studies of sufficient 
classrooms or sufficient in -depth examination of a cross-section of class looms. 

The study from which the substance of this paper was drawn invoJ.ved 
an ex t en s i ve c u r r i cu 1 vim e va 1 u a i on j.^ r o j e c t c h a r a c t c r i s c d b y ma n y of t he alj o v 
faetif^rs. Six hviulrr/i rjfty te:'jl crn report.',! on fr^'C i r r . ■ neitahiinv, to :ni 
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innovative iiiqu : ry OT-iontf-d Junior Higli School Science Curriculuut, uiiile 43 
clasps roOiPS and 1163 sLudcnts v;crc obscrrved and touted in <'m in-dcpth study 
of the curr Icu-l wri in c-j^crn t. i un and nfiuU^nt oul:co::if'S on cognitive, percoption, 
attitude, and inlrcrest mca^^urcs. 

The objective of the paper v/as to describe the conception, operation- 
alisation, validation, and ro].e of an Arbitrary Iiuplenientation Scale in an 
ex post, facto curriculum evaluation model vs?hicli facilitated the assessment of 
factors related to this curriculum and situation^ It was concluded uhat the\j 
use of an implenientation scale can enhance an evaJ.ua tive study. This has/ 
been underlined by Charters and Jones (1973) , subsequent to the conclusic:! 
of this study, who say that V7hile it is becoming standard practice to utilise 
considerable resources for conducting evaluation of student outcomes for 
"experimental" and "'control" schools or classrooms, it "is not the standard, 
practice in evaluation studies to describe, let alone measure, how the program 
in "experimental" and "control" situations actually differ from one another - 
or even certify tliat they do!" Accordingly, they specify four levels of 
reality that may exist: firstly, Ins tltuti c nal Corimil tment , that is the formal 
announcement or introduction of the "innovation" by the administration; 
secondly, the Structu ra l Con t e x t level, which includes the changes in formal 
arrangements and physical conditions (e.g, making the necessary equipment and 
facilities available); Role Performance, the third level of implementation, 
involves the actual necessary behavioral changes in teachers; while the fourth 
level, Lea v n i n g A c: x i v 1 1 1 e s , involves the intended classroom transactions, 
which, hopefully, v.-J ] J enable r. rudu^nts to reach the intended learning out ceres 
ol ll..- ^ m;. i/iuuJ ii;;,. 

This evaluative study assumed that Level I had been attained by the 
foriaal announc. 5;:cnt c^f t'ne Trovi'iicial D'.'iJ.'.rtnont of TcUication (1968), but, cf 
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course cliir; does not necessarily mean any sl^yiif i cant: change in classroom 
transactions I Tiie scale, therefore, included elements of Structural Context , 
Role Perf ormrincc ^ and L earni n ^; Ac: t :l vi t J e s . Some items wliich could fall under 
these categories were included in data collection but excluded from the 
actual AIS scale so that the process of implementation could be examined 
during data analysis. In essence, then, the AIS attempted to quantify on 
a c on t in u on s s c al e , as oj^posed to tlie f our l evel approach of Charters and 
Jones, the degree of impJ.ementati on of an innovation v/ithin IndivlduaJ class- 
rooms on a Province-wide basis. 

The results of th.is attempt to develop and utilise a valid ATS, which 
minimises the i)ossibility of what Charters and Jones call '\Tppraising a 
non-event^', are encouraging. 

Undoubtedly, this initial effort in the utilisation of an AIS has not 
exhausted all of its ]>otcntial uses, nor, indeed displayed exemplary devel- 
opment and validation procedures, Furth:>r research is needed within the 
realm of implementation of innovations to assist in identifying items for 
less crude AIS scales. 

While it is realised that efforts must be made tu provide inbuilt 
evaluation procedures at the outset of curriculum development and innovation, 
it is hoped that perhaps the procedure used in this study might provide a 
basis for other pragmatic assessments of curricula operating in similar 
situations of uncoordinated change, v/hether as the result of poorly coordinate 
central initiative or v;id(^snread grass-roots initiatives. 
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APPENDIX 



ERIC 



ITKMS usi:d in t;ii; akuttrais^y n-n'iA:i-n:NTATroN scau:; 



.1 . Is .scicMice Liu.: subject you foci most cap^ible of teaching? Yes D NoD 

2. Did you have Iho oj)port unity to attctncl in--Gcrvico session(s) on Division TII 
S'-ionce? (SfM rial vorl:sh(>p, institute, convonLion) N.B. Do not inclucie 
UnLvorsity credit classes. Yes (J NoD 

3. ir *yes', how many? 



EsLip.Mlc the arKiunL of cl:.\ss time uevoli-d to each of the foliowUv.; types of 
aciivitics diu'.in[', tliu course of a year. (Note: Times do not nc^ed to add up 
to 100/.) 



^4. (a) LecLur-'.n'j 

and tl t* Hivon s 1 1" a I i « ) i\ 



71 and 

^ Iz^hi 1^-20% 2_ij<n)% 31-40% /iL-J-ir^l PJs.^.^i. P.LzI^ ' lisL^ 

□ □ □ □ □ n □ n □ 



3. (h) Studenls fcMi- 
duc t i nj;^ i nvcs i .1- 

□ □ □ □ □ □ □ n □ 



Indicate the emph-^sis you ^; I vo the folJo\;ing iljethods of evaluation in arrivin[^ 
at a final mark in Division 111 Sciences. (]'ercentae,es do not need to add up 
to 100Z.) 22 '-ind 

91 i7.^J^i 11 -20% 21-307 . 31-A0% 41-50% _51j:60Z 61jJ'0% inore 

6. Studt fit lai) and 

project work. □ □ □□□□□□□ 

7. Formal exam- 

'■'•-'LLons □□□□□□□□□ 

8 Indicate yoiu' level ol aj^recinent vi th the philoso])hy of Division III Science. 

Low ■ Hij^i 
1 2 3 ^4 "5 

llecans,'.' of v.':ri(nis factt->rs (r.uch. as pr l'-sc rvl ce and ln~ser\' i e education, 

f ac i I • t it'S . ecr.) I ftu-l th/tt for thu ^;rados 1 teach my lev't.Lo of competence 

iu til*.- lo How 1* n^', aspects of the Divisi<'i\ LT.l Ceiei^ce Prugrat.i are as ft)] lt.«.vs . 

Low H i gl 1 

9 . U p» d e rs t aT> d i n t e Division 111 S c i p c n 

phi losojihy p\)c\ ob ject Ives ] 2 3 A 5 

10. f:i;MK. ,.r (he ^ .nt ^-w i / 3 ^ 

IJ. Knov/led;.',e of ma t'- r i a 1 fi and efj u 1 1 :i'c/i I r<**jtjJrr'd 1 y t ^ 

£PJ|^12. A!)ility to Uiach t!ip ;)roe.ram 1 2 3 A 
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I-;Ut» i]\c J i^] ! ov.' i .11;; in li'ti-^-. o\' I ho i r UL^iM'uliiess in lielpin^^, you Co .i np] eraon t: lIk^ 



13. (k)()jJ .space iaciJitios 

.1 '^4 . Ad ru 1 1 J : ; I r n I: i. V c ju t [ > p o t 

13. Bt;l irf in ihc plii 1 ;n:o;.by of the p»'0[;,rnni 



]{) 
17 

.1 S 
.19 
2U 
21 
22 
2 3 



l,n ( 1 . ( ^ i cqu i [JiuLM. t. » 

I u.'ulcqua l.e riM"d ciic e and l ihr.-iry resources. 

Tn.^dt'j]iw» I c prc'--sf ' r vi cc cducr. tion. 

I'uor j'pai're faci 1 I t ics 

Adii; 1 : 1 i s t' r a L i V( ? c o ik'; I ro J. n L s 

Ad V : 1 r. 1 1 '.d c n I r c . + c I: i on 

L.'urk I'f srlio.^^l Ijurrid suppp'"!. 

Irn'k belief in phiJ.i'Sophy t)f prui-i-t'. ... 



.1 
1 
1 



Koi 



A Great 



Soiiieu'bat" 






2 3 




5 


2 3 


4 


3 


2 3 




3 


nd^'retl su(xtr 


ssful 




t:eac.b . 
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Crp at 








2 3 
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2 3 
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2 3 
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2 3 
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2 3 
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2 3 
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2 3 
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2 3 


*■» 


L' 
.) 



VfMij- opinli'i) ru the i'c^ 1 mo in s tat f.-nien t vith reforoncG to Ujvj.siun 1.11 



^jf i (Mici.- I'ro^i aiu for ^-radcLi 7, B, and- 9, 



24. Tlio n];proacli nscnl i.n t.he Division IJI 

Science Progra:;i:i for [grades 7, 8, and 9 iv^ 

niiK^h the saiuc- as fradirional GcJence 

ti.'dch i ng . * * , 

23. A ttacher in eacli parade ^^hould stay very 
c.lc^se to tlic 'ni'jjr-ot matter for that 



Stron Y ^ ^ ' ^ ^ 

Pis a^rco Unde cjjic'd Ajj.i'cc 



3 4 



grade: 



27, 



One ir.Ljor cinphasir-: in the pro;.M;ai:i is I:o 
iisj^ rialnr idea.': in rfilatinj.'; diffin'Mit 
parts fjf tlie conrr^e 



The ova lua Lion of student?: i:un-.t be based 
lar^'/":.l}* on cog.aitivo ou tetanies 



3 
5 



28. Gene --a 1.1 y J tiie 1):. vision Hi Science l^:o[;,ram 
h ;.u; i I a d 3 i t L 1 e 1 f e c t i n e 1 1 a n i n ^ Kiy 
tc. i' hi n;'. 



3 4 
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The Cur r j cuium 

The general aspects of the program were to include the following 
charac: terx^d t ics : 

• The pro gram shoulr] h ave _a unitary flavour for^Jlh^ three grades 
and provicie a three-year cycle . It v;as hoped tliat the grade 9 year would 
provide a capping effect upon the previous two years and that a student 
would touch upon tlie main content areas of life, earth and space or physical"^ 
science during his gra.de 7, 8 and 9 school career. 

2 , Th e ]-) r o g r ara ^1 lould b e bui 1 1 arou nd conce p t u a l and be h a vioral 
schem es in scien ce , Recent thinking in science education at that time seemed 
to suggert that a more fruitful way of organizing curricula vjas to utilize 
major ideas in science rather than subject -matter content. In addition, cer- 
tain behavioral themes were identified which would serve to give coherence 
to student experience. These themes were to provide an overall structure to 
which the three content areas V7ould contribute. Specific concepts v/ithin 
each subject area were identified for possible study at the three grade levels. 

^ * The scien ce program should em p hasize student involvem e nt with 
materia ] s a nd the investigations should provide for an inquiry approach to 
learning . The implications of this facet of the program were that equipment 
and materials should be available. It was noted that the inquiry and inves- 
tigation approach would require a major change in role for many teachers* 

Following a discussion of the research proposal with origina]. and 
current members of the Provincial Science Committee that developed the cur- 
riruluir., they v'^.t tended to : » ' x-l v'/'; ''\^' Lite ohj^-ri'^v-s.'^ 

^ Alternate approaches to grade 9 science were permitted (Space Science or 
Introductory Physical Science) as both were consistent with the philosophy 
of the junior hi^h scli no 1 ncxence. pro'/. ::am an«:1 a nntnber of tericlierij had 
prefe red IPS during curric j].ri:i durolopMC'iit . 

The clarification of objccLivcs" detailed here is properly part of the 
evaluation model and researcli procedures discussed later, but is appropriate 
I O for inclusion here. 

ERIC 



1. To develop in the sLud-uL an apprec ira i on for, and an iiitoresL 
in science. 

2. To challenge the student to think and reason throu^^h sciicntiric 

study . 

3. To foster a spirit of inquiry. 

4. To develop an understanding and appreciation of th»e methods by 
which scientists discover new knowledge. 

3. To dfveloj) a deeper insight into the interrelar.ion.ships that 
exist in scjc-nce. 

6. To develoj^ and hroaden the student ^s understanding of the big 
ideas of science. 

7. To help the student acc}i.:ire sonic of the skills arid processes 
of science. 

It was believed that thest^ objectives could be achieved only in 
classroor.s meeting certain mini mum requirements. These included: 

1- Adequate facilities and the addition of science equipmenr where 

needed . 

2. The addition of new resource books and teaching materials. 

3, A change from traditional classroom transactions. The teacher's 
role was to stinmlate and challenge the student to think and to provide an 
atmosphere of freedom of operation. He was to provide support for inquiry 
and manipulation and investigation of materials, giving the student the 
opportunity to pursue problems and exercise some auconomy in interpreting 
results. Compared to the more traditional directive and expository role, a 
teacher was intended to be open, ncn-dirccti\,'e and to act as a resource 
person and guide to learning',. 

4 . I n - s ^ ■ rv i c; c- n d n c a t i o n to a ^ ; i r-; I L i ■ a c h a r : ; t o c or. rL'\.cui} : ; n c I i Im p 1 c 
the programme . 
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TAl'.LI- VT. 

Mli'AK';; A-sD STANDAia) i.iEVTA'QON'S 
0)^ A) S ] Ti;;lR 
rOR ALL RESP().\-L1:k'TS 



1 1.48 

2 1.50 

3 2.26 
^ 3.93 

5 A. 04 

6 3.97 

7 3.82 

8 3.50 

9 3.27 

10 3.71 

11 3.43 

12 3.39 

13 2.67 
1-^ 2.92 

15 5.53 

16 3.03 

17 2.97 
1« 2.97 
19 3.00 
^0 1.76 
21 2.08 



SPANDAKD DEVIATION n 



0.50 

0.50 

1.41 

1.56 

1.76 

1.57 

2.05 

0.92 

1.08 

0.92 

1.02 

0.87 

1.24 

1.28 

1.00 

1.25 

1.22 

1.22 

1.30 

0.97 

1.02 



603 

585 

606 

601 

556 

567 

530 

536 

599 

603 

604 

599 

571 

584 

567 

587 

590 

572 

580 

569 

583 
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Continued 



22 1.90 1.02 563 

23 1.96 1.00 569 

24 2.01 0.9S 585 

25 2.22 1.16 590 

26 3.77 1.01 578 

27 2.95 1.01 568 

28 2.46 1.12 568 
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TABLE VI. I. 



I'.EAN AND STA.\'DARD DEVIATIONS OF AIS ITEMS 
FOR TOj' A.\D r.OTTOM SCOKLNG HALVES OF RESrONDENTS 



ITEM 




TOP 






BOTTOM 




MEAIv! 


S.D 


N 


MEAN 


S.D 


N 


1 


1.26 


0.44 


292 


1.70 


0.46 


90*^ 
z.yj 


2 


1.29 


0.46 


305 


1.71 


0.45 




3 


2.39 


1.47 


211 


1.93 


1. 20 






3.45 


1.62 


303 


4 .42 


JL • D / 


o o o 


5 


4.78 


1.78 


290 


3.24 


JL • J J 


zoo 


6 


4.52 


1.64 


303 


3.34 


1 9 9 


0 /I /. 


7 


3.15 


1.77 


275 


4.53 


9 DQ 


o c c 


8 


3.82 


0.79 


280 


3.15 


n Q9 


25d 


9 


3.75 


0.90 


303 


2.78 


1 09 


0 O 


10 


3.99 


0.83 


305 


3.42 


0 


9 0 Q 


11 


3.87 


0.86 


305 


2.99 


0 QO 
^ ' y J 




12 


3.71 


0.78 


302 


3.07 




O O "7 


13 


3.19 


1.17 


288 


2.14 


1 Oft 


*1 o o 

283 


14 


3.35 


1.21 


298 


2.48 


1 90 

JL • Z\J 


O O (I 

zoo 


15 


3.93 


0.90 


299 


19 
J • JLZ 


0 , 93 


275 


16 


2.59 


1.21 


296 


3.47 


1.13 


291 


17 


2.62 


1.19 


297 


3.33 


1.15 


293 


18 


2.43 


1.16 


286 


3.50 


1.04 


286 


19 


2.60 


1.28 


290 


3.41 


1.20 


290 


20 


1.56 


0.87 


289 


1.95 


1.03 


280 



Coiitiinued 



Coat in' -d 
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21 
22 
23 
24 
25 
26 
27 
28 



1.78 

1.66' 

1.58 

1.65 

2.06 

3.87 

2.79 

2. 10 



0.83 
0.92 
0.85 
0.82 
1.09 
1.00 
1.07 
1.08 



292 
,283 
286 
294' 
299 
290 
286- 
286 



2.38 
2.14 
2.35 
2.37 
2.38 
3.67 
3.12 
2. 84 



1.10 
1.05 
1.00 
0.99 
1.21 
1.02 
0.93 
1.03 



291 
260 
283 
291 
291 
288 
282 
282 
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tai^ll: viii 

coKKKLAi io:; ]>i-)'wkk:; ats rrK>;s aud total ais scoiuis 



ITF.M C0I;RELATI0N 

1. Is science the subject: you feel most capable 

of ^ eachii^r, ' , --SO 

2. Opportunity to attend J n -service sessions? -.47 

3 . How many? .22 
A, Estimate liiiio spent lecturing and demons tratin^^^ • .37 

5. Estimate time spent v:hen students were conductxnp, 

invx^s tigatlons .56^ 

6. \\o\'7 mucl\ dues student lab ^ud project \.'ork feature 

in final evaluation .43 

7 . Hov; much docs f orma 1 ex ain in a t i on and project v;o r k 

ft-ature In final evaluation -.43 

8. Level of ae,rcemcnt witii philosophy of curriculum .47 

9. Understanding of pliilosophy and objectives .56 

10. ICnovledge of content .42 

11. Knov\'ledge of materials and equipment required .32 

12. Ability to teach program .47 
Ratings of usc^ful^-.ss in implementation: 

13. Good space facilitief; .5]. 

14. Adminis tra I i\ e support .42 

15. Belief in i lof-ophy of program .31 
R/j 1 1 n r s of hi n d r a n c e i n i n 1 cv\ c n \: : ^ t i o n : 

16. Lack of equipment -.46 
17 Inadequate reference and li brary resources -.38 
in. Juitdi.qu 1 1 e i^rer.s'Mfv.' ce vducat.ion -.33 



Cout inued 




Cr^'i L ini= i 



19. r(»or Bpaco Sac :i 1 i t i t\s -.39 

20. Adni i n Lsl ra I Ivu cons Lrai tits -.26 

21. Advori^c stuclcMiL rennet ion -.38 

22. Lack of ncliool board support' -.27 

23. Lack of b^.'.l.icf .in plii. 1 osui^li)' of ])ro^r£in) -«.47 
Op i n iony : 

24. Kcw pro[;rar) cauic ,ks L rad i I i on<'. 1 pioi'.rciin ? -.43 
2^. Teacher should .-.tick rlost->ly to j:nbicct: v.]C^ttcA''{ -.18 

26. ^^ajor c ,ipl)asir, is to irn- "bi.[^," idca^.? --.14 

27. Kva Iiiat i oa based Jar^'/'ly on c o^;n^^' ^ "^'^^ uut:CL-;iu\s ? -.19 

28. Nc'ic progr:un has had little effect Iw ehan^^.iii- 

my teaching; -.35 



